Skip to content

fix(ingestion): report the import environment from the heartbeat asset - #838

Merged
jirhiker merged 1 commit into
stagingfrom
fix/ingestion-import-path
Aug 18, 2026
Merged

fix(ingestion): report the import environment from the heartbeat asset#838
jirhiker merged 1 commit into
stagingfrom
fix/ingestion-import-path

Conversation

@jirhiker

Copy link
Copy Markdown
Member

database_connectivity fails in Dagster+ with ModuleNotFoundError: No module named 'db', while the code location loads and ingestion_heartbeat runs.

Two attempted fixes have not worked — inserting the repository root from
automated_ingestion/__init__.py, and setting PYTHONPATH=/opt/dagster/app.
So the assumption behind both is wrong somewhere I cannot see from outside the
container.

What has been ruled out

  • db/ is missing from the image. It is not. I rebuilt the image locally
    from the same template and context (context: ./, .dockerignore excludes
    only .venv) and /opt/dagster/app/db is present.
  • The code is wrong. In that same image, find_spec('db') resolves and
    from db.transducer import TransducerObservation succeeds.
  • A missing dependency masquerading as db. In the image, an absent
    dependency names itselfNo module named 'sqlalchemy', not 'db'.

The remaining explanation is that in the step process, automated_ingestion
resolves from somewhere db does not. My local check ran python -c, which
puts the working directory on sys.path; Dagster runs a console script, where
it does not. That is a real difference and I cannot see which paths the step
process actually has.

What this adds

ingestion_heartbeat now reports cwd, the resolved app root and its
contents
, whether db and domain are findable via find_spec, and the
full sys.path. It needs no credentials and no database, so it reports even
when everything else fails.

Materializing it answers the question directly instead of another round of
inference.

PYTHONPATH is also added to the setup script's storage phase — harmless if
it turns out not to be the cause, and correct regardless.

🤖 Generated with Claude Code

Inserting the repository root from automated_ingestion/__init__.py did not fix
the ModuleNotFoundError for db in a step process, so the assumption behind that
fix was wrong somewhere I cannot see from here.

PYTHONPATH=/opt/dagster/app makes the app root importable regardless of how a
process was launched, rather than depending on the package having been imported
first or on the working directory being on the path.

The heartbeat asset now reports cwd, the resolved app root and its contents,
whether db and domain are findable, and sys.path. It needs no credentials, so
it reports even when everything else fails -- which is what a diagnostic asset
is for. If PYTHONPATH does not resolve this, its metadata says why.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jirhiker
jirhiker merged commit 5ccb0d6 into staging Aug 18, 2026
9 of 10 checks passed
@jirhiker
jirhiker deleted the fix/ingestion-import-path branch August 18, 2026 23:25
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Coverage

79.27% total — gate is 75%.

Coverage for the Python files changed in this PR
Name Stmts Miss Cover Missing
automated_ingestion/defs/assets/heartbeat.py 15 2 87% 55-56
TOTAL 15 2 87%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant